Papers with policy adherence
Beyond IVR: Benchmarking Customer Support LLM Agents for Business-Adherence (2026.eacl-industry)
Copied to clipboard
| Challenge: | Existing benchmarks focus on tool usage or task completion, overlooking an agent’s capacity to adhere to multi-step policies, navigate task dependencies, and remain robust to unpredictable user or environment behavior. |
| Approach: | They propose a benchmark to assess policy-aware agents in customer support using a dynamic-prompt agent and a static-promped agent that explicitly models policy control. |
| Outcome: | The proposed benchmark assesses agent's ability to adhere to multi-step policies, navigate task dependencies, and remain robust to unpredictable user or environment behavior. |
Towards Safety Reasoning in LLMs: AI-agentic Deliberation for Policy-embedded CoT Data Creation (2025.findings-acl)
Copied to clipboard
Tharindu Kumarage, Ninareh Mehrabi, Anil Ramakrishna, Xinyan Zhao, Richard Zemel, Kai-Wei Chang, Aram Galstyan, Rahul Gupta, Charith Peris
| Challenge: | Safety reasoning paradigms require high-quality policy-embedded chain-of-thought datasets . generating such data through human annotations is prohibitively expensive . |
| Approach: | They propose AIDSAFE: Agentic Iterative Deliberation for Safety Reasoning . AIDS AFE leverages multi-agent deliberation to iteratively expand reasoning on safety policies . |
| Outcome: | The proposed model improves policy adherence and reasoning quality while maintaining acceptable utility and over-refusal accuracy. |
CAR-bench: Evaluating the Consistency and Limit-Awareness of LLM Agents under Real-World Uncertainty (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for Large Language Model (LLM) agents focus on task completion under idealistic settings but overlook reliability in real-world, user-facing applications. |
| Approach: | They propose a benchmark to evaluate consistency, uncertainty handling, and capability awareness in multi-turn, tool-using LLM agents in an in-car assistant domain. |
| Outcome: | The proposed benchmarks evaluate consistency, uncertainty handling, and capability awareness in multi-turn, tool-using LLM agents in an in-car assistant domain. |